Transformer-based Vulnerability Detection in Code at EditTime: Zero-shot, Few-shot, or Fine-tuning?

Software vulnerabilities bear enterprises significant costs. Despite extensive efforts in research and development of software vulnerability detection methods, uncaught vulnerabilities continue to put software owners and users at risk. Many current vulnerability detection methods require that code snippets can compile and build before attempting detection. This, unfortunately, introduces a long latency between the time a vulnerability is injected to the time it is removed, which can substantially increases the cost of fixing a vulnerability. We recognize that the current advances in machine learning can be used to detect vulnerable code patterns on syntactically incomplete code snippets as the developer is writing the code at EditTime. In this paper we present a practical system that leverages deep learning on a large-scale data set of vulnerable code patterns to learn complex manifestations of more than 250 vulnerability types and detect vulnerable code patterns at EditTime. We discuss zero-shot, few-shot, and fine-tuning approaches on state of the art pre-trained Large Language Models (LLMs). We show that in comparison with state of the art vulnerability detection models our approach improves the state of the art by 10%. We also evaluate our approach to detect vulnerability in auto-generated code by code LLMs. Evaluation on a benchmark of high-risk code scenarios shows a reduction of up to 90% vulnerability reduction

Cross-Device Taxonomy:Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices

Designing interfaces or applications that move beyond the bounds of a single device screen enables new ways to engage with digital content. Research addressing the opportunities and challenges of interactions with multiple devices in concert is of continued focus in HCI research. To inform the future research agenda of this field, we contribute an analysis and taxonomy of a corpus of 510 papers in the cross- device computing domain. For both new and experienced researchers in the field we provide: an overview, historic trends and unified terminology of cross-device research; discussion of major and under-explored application areas; mapping of enabling technologies; synthesis of key interaction techniques spanning across multiple devices; and review of common evaluation strategies. We close with a discussion of open issues. Our taxonomy aims to create a unified terminology and common understanding for researchers in order to facilitate and stimulate future cross-device research

The Structure of Consensus Building in Online Design Discussions

While consensus building is inherently complex, the remote and asynchronous nature of online discussions, coupled with the large number of participants further intensifies the complexity. To better understand the nature of online consensus building, we first interviewed designers and developers contributing to design discussions in open source projects to understand the key factors in online consensus building. We then collected a large data set from an online design Web site and used regression analysis to further test how these factors affected consensus building. Our analysis showed that participants who were more experienced in the community facilitated consensus building. In addition, participants who had previous social interactions with each other would more likely reach a consensus in future discussions. We also found that even though the number of participants in each discussion was often large, small-group personal interactions did emerge, which significantly predicted the likelihood of reaching a consensus. Implication to design of interfaces that facilitate online design discussions is discussed.unpublishednot peer reviewe

VICPAM: An Interactive Analysis Tool for Visualizing Interaction and Communication Patterns in Multiple Display Environments

Multiple Display Environments (MDEs) facilitate collaborative activities that involve the use of electronic task artifacts. Supporting interactions and infrastructures have matured in recent years, allowing researchers to now study how the use of MDEs impacts group work in controlled and authentic settings. This has created a need for tools to understand and make sense of the resulting interaction data. To address this need, we have designed and developed a new interactive analysis tool called VICPAM. Our tool consists of several components: (i) A spatial map view, which gives a 2D overview of all users??? activities; (ii) a time-aligned view, which shows users??? activities and the duration of each activity; (iii) the time slider, which can be used to adjust the period of time under study; and (iv) a video of the session synchronized with the timeline. The tool can reduce the effort necessary to analyze and make sense of users??? interaction data in MDEs

Consensus building in distributed design discussions

With the rapid growth of open source and other geographically distributed software projects, more User Interface (UI) design discussions are occurring online. These discussions typically unfold through interactive web forums where multiple stakeholders generate and debate different design proposals. One important challenge is bridging the different perspectives and needs of stakeholders and deciding on a design proposal that will be implemented and integrated into the product distribution. Failure to reach agreement can cause discussion participants to loose interest and therefore the product would not be improved. However, the existing discussion platforms do not provide any mechanisms to aid the process of reaching a decision. Prior work has analyzed distributed design discussions from many perspectives, but my PhD dissertation brings consensus building as a new theoretical lens to study and support decision making in design discussions. This has allowed my work to make three contributions. First I contribute empirical knowledge showing that nearly half of the discussions do not reach consensus and knowledge of the content, process, and social factors that affect consensus building in distributed design discussions. Second, to address the inability to reach consensus, I developed a novel browser extension called Procid that provides interaction and visualization features for bringing consensus strategies to distributed design discussions. Key features include the ability to organize discussions around ideas, register and visualize support for or against ideas, and define criteria for evaluating ideas. It also applies interaction constraints promoting best practices of consensus. Finally, I present the results of two evaluations of Procid. The first collected perceptions of the tool from members of a large and successful open source community (Drupal.org) for their own discussions. The second compared how Procid affects a distributed design discussion relative to the current discussion platforms. Results of both studies showed that users perceived Procid as more effective for consensus building than the existing platforms

Consensus Building in Distributed Technical Discussions

The issue management systems in open source software projects contain different categories of discussions, but they all share the goal of reaching consensus on solution proposals. In this paper, we examine the consensus building process in distributed discussions of technical issues in one mature open source software community. Our analysis shows that providing a concrete solution in the form of a patch implementation is most predictive of reaching consensus in technical discussions. This is in sharp contrast to prior work on consensus building in distributed UI design discussions which showed that having participants with more experiences and prior interaction histories are more predictive of reaching consensus. Our results highlight that consensus building depends on the nature of the issue. In technical issues that tend to be more driven by objective evaluations providing more patches promotes consensus. However, in UI design issues that tend to be more driven by subjective evaluations, having more experience participants helps discussions to reach consensus.unpublishednot peer reviewe

Open Source Interface Politics: Identity, Acceptance, Trust, and Lobbying

A study of the Drupal open source project shows the rather problematic status of usability designers with respect to the larger developer community. Issues of power, trust, and identity arise and affect the way that usability recommendations are acted on or ignored. A political view of these aspects can help in interpreting the situation. We found that making a straightforward case for a particular interface design can be insufficient to convince developers. Instead various additional lobbying strategies may be employed to build up a quorum of support for the designunpublishe

RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shot

Performance bugs are non-functional bugs that can even manifest in well-tested commercial products. Fixing these performance bugs is an important yet challenging problem. In this work, we address this challenge and present a new approach called Retrieval-Augmented Prompt Generation (RAPGen). Given a code snippet with a performance issue, RAPGen first retrieves a prompt instruction from a pre-constructed knowledge-base of previous performance bug fixes and then generates a prompt using the retrieved instruction. It then uses this prompt on a Large Language Model (such as Codex) in zero-shot to generate a fix. We compare our approach with the various prompt variations and state of the art methods in the task of performance bug fixing. Our evaluation shows that RAPGen can generate performance improvement suggestions equivalent or better than a developer in ~60% of the cases, getting ~39% of them verbatim, in an expert-verified dataset of past performance changes made by C# developers

The Need for Richer Refactoring Usage Data

Even though modern Integrated Development Environments (IDEs) support many refactorings, studies suggest that automated refactorings are used infrequently, and few developers use anything beyond Rename and Extract refactorings. Little is known about why automated refactorings are seldom used. We present a list of challenging questions whose answers are crucial for understanding the usability issues of refactoring tools. This paper argues that the existing data sources—Eclipse UDC, Eclipse refactoring histories, version control histories, etc.—are inadequate for answering these questions. Finally, we introduce our tools to collect richer usage data that will enable us to answer some of the open research questions about the usability of refactoring tools. Findings from our data will foster the design of the next generation of refactoring tools.published or submitted for publicationis peer reviewe

A Compositional Paradigm of Automating Refactorings

Though modern IDEs have supported refactorings for more than a decade, recent studies suggest that programmers greatly underuse such tools, especially for complex refactorings. Complex refactorings affect several methods or classes and tend to be tedious and error-prone to perform by hand. To promote the use of refactoring tools for complex changes, we propose a new paradigm for automating refactorings called compositional refactoring. The key idea is to perform small, predictable changes using a tool and manually compose them into complex changes. This paradigm trades off some level of automation by higher levels of predictability and control. We show that this paradigm is natural, because our analysis of programmers’ use of the Eclipse refactoring tool in the wild shows that they frequently batch and compose automated refactorings. We then show that programmers are receptive to this new paradigm through a survey of 100 respondents. Finally, we show that the compositional paradigm is effective through a controlled study of 13 professional programmers, comparing this paradigm to the existing wizard-based one.NSF CCF 11-17960; DOE DE-F02-06ER25752unpublishednot peer reviewe